An Audio-Visual Imposture Scenario by Talking Face Animation

With the start of the appearance of PDA’s, handheld PC’s, and mobile telephones that use biometric recognition for user authentication, there is higher demand for automatic non-intrusive voice and face speaker verification systems. Such systems can be embedded in mobile devices to allow biometrically recognized users to sign and send data electronically, and to give their telephone conversation a legal value. The European project “Secure Contracts Signed by Mobile Phone” (SecurePhone) aims at developing such technology on a 3G/B3G enabled PDA. One of the risks that a speaker verification system could face is its liability to imposture. With the current communication infrastructure lacking strong user identification, impostors aware of legal transactions can interfere and be engaged in a telephone conversation so as to alter or replace the true conversation, or even initiate a conversation and impersonate in it another person. To combat imposture, it is necessary to study imposture techniques and scenarios. In this paper, we implement a system that allows an impostor to start and lead an audio-visual telephone conversation, and sign and exchange data electronically on behalf of another person. During the conversation, audio and video of the impostor are altered in a way as to mimic the other person’s voice and face. On the speech side, there exist processing techniques exploitable by impostors to reproduce the voice of an authorized client. In particular, speech segments obtained from client's recordings can be used to synthesize new sentences that the client never pronounced. We will explain how a very-low bit-rate speech coding system, such as the ALISP-based one, can be adapted to serve forgery purposes, transforming any input speech into client's voice. On the human face side, the imposter’s talking face is detected and facial features are extracted and tracked. Lip movements are used to animate a synthetic talking face (Greta). The texture of the impersonated face is mapped onto “Greta” and coded for transmission over the phone, along with the synthesized voice. Audio-visual coding and synthesis is realized by indexing in a memory containing audio-visual sequences. Stochastic models (coupled HMM) of characteristic segments are used to drive the search in memory.